Skip to content

[bug](GC)the issue of incorrect disk usage#18397

Merged
morningman merged 5 commits into
apache:masterfrom
chenlinzhong:fix_disk
Apr 8, 2023
Merged

[bug](GC)the issue of incorrect disk usage#18397
morningman merged 5 commits into
apache:masterfrom
chenlinzhong:fix_disk

Conversation

@chenlinzhong

Copy link
Copy Markdown
Contributor

Proposed changes

disk used capacity is incorrect:
DataUsedCapacity+AvailCapacity !=TotalCapacity
image

Problem summary

some invalid rowset rowset not be deleted for ever due to the bug:
image
when key_may_exists() return true not means it is real exists

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@chenlinzhong chenlinzhong requested a review from morningman April 5, 2023 03:17
@chenlinzhong chenlinzhong changed the title [bugfix] Fix the issue of incorrect disk usage [bug] (fix) the issue of incorrect disk usage Apr 5, 2023
@github-actions

github-actions Bot commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@chenlinzhong chenlinzhong changed the title [bug] (fix) the issue of incorrect disk usage [bug] (GC) the issue of incorrect disk usage Apr 5, 2023
@chenlinzhong chenlinzhong changed the title [bug] (GC) the issue of incorrect disk usage [bug](GC)the issue of incorrect disk usage Apr 5, 2023
@chenlinzhong

Copy link
Copy Markdown
Contributor Author

run all

@morningman morningman self-assigned this Apr 6, 2023
@chenlinzhong

Copy link
Copy Markdown
Contributor Author

run p0

1 similar comment
@chenlinzhong

Copy link
Copy Markdown
Contributor Author

run p0

@chenlinzhong

Copy link
Copy Markdown
Contributor Author

run beut

@chenlinzhong

Copy link
Copy Markdown
Contributor Author

run feut

@chenlinzhong

Copy link
Copy Markdown
Contributor Author

run p0

@chenlinzhong

Copy link
Copy Markdown
Contributor Author

ru beut

@chenlinzhong

Copy link
Copy Markdown
Contributor Author

run clickbench

Comment thread be/src/io/fs/file_system.cpp Outdated
Comment thread be/src/olap/rowset/rowset_meta_manager.cpp
Comment thread be/src/olap/tablet.cpp Outdated
@hello-stephen

hello-stephen commented Apr 6, 2023

Copy link
Copy Markdown
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 36.12 seconds
stream load tsv: 443 seconds loaded 74807831229 Bytes, about 161 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 72 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230407160549_clickbench_pr_126080.html

@github-actions

github-actions Bot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

3 similar comments
@github-actions

github-actions Bot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions

github-actions Bot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions

github-actions Bot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@chenlinzhong

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman morningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 1616783 into apache:master Apr 8, 2023
morningman added a commit that referenced this pull request Apr 10, 2023
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up #18397
3. Add KW_HOSTNAME to keywords region, follow up #17329
4. Fix tvf not working with pipeline engine, follow up #18376
morningman added a commit that referenced this pull request Apr 11, 2023
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up #18397
3. Add KW_HOSTNAME to keywords region, follow up #17329
4. Fix tvf not working with pipeline engine, follow up #18376
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
…#18485)

1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up apache#18397
3. Add KW_HOSTNAME to keywords region, follow up apache#17329
4. Fix tvf not working with pipeline engine, follow up apache#18376
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
…#18485)

1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up apache#18397
3. Add KW_HOSTNAME to keywords region, follow up apache#17329
4. Fix tvf not working with pipeline engine, follow up apache#18376
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants